Captions provided by @chaselfrazier and @whitecoatcapxg.
      "Using Mapillary to Improve OpenStreetMap."
      By: Johan Gyllenspetz.
      >> Good afternoon, everyone.  Please welcome Johan for the next talk.
      [Applause]
      >> Does this work?
      >> Yes.
      >> Does it work now?
      >> Yes.
      >> Sorry.  Maybe this is a good way?  Okay.  Do I usually -- do I need a microphone?
      >> Yes.
      >> All right.  Okay.
      So, first of all, thank you, all, for coming like this later afternoon to come take a look at what we do at Mapillary.  My name is Johan with a heart Swedish last name, Johan Gyllenspetz, I'm a Swede that has emigrated to the states for sunshine and love.
      So currently in Los Angeles and I'm the lead architect for a company called Mapillary, and we've been running for about two and a half years now.  And in this presentation, I want to talk a little bit about how to use Mapillary to improve OpenStreetMap.  But I'm going to do it in pretty much three sections.  First, I'm going to talk a little bit about the Mapillary S.  And then I'm going to talk about our open platform and how to develop against that and then I'm going to finish off with a couple of examples of how people have used our platform to integrate Mapillary into the different OpenStreetMap tools.
      So at this point you're probably asking yourself what is Mapillary?  Well, Mapillary could be -- I don't know the English word for this but combined in one sentence.  Mapillary is a service for crowd so you first street level photos.  Ever since we started Mapillary, we had these single thoughts about how to crowd source imagery.
      First of all, we think that photos should be collected collaboratively in the open -- in the open with an open license.  So all of our interests are license with open license.  They're actually created comments shared by share life.
      Anyone should be able to contract and use regardless of what the motivation is for uploading the images to this big image repository that we're creating.  And at least but not the least important is that we all come from the computer vision part, and we thank if we're collecting this vast amount of data that images are, we do think that we usually use computer vision to make sense of this data so that you can use this data that you have extracted to make better maps.
      So this is Mapillary.  This is our current coverage in the world, you can see that Europe is pretty well covered.  The coast in the United States are starting to get pretty covered too.  All of the -- at this point I should probably say green things on here is where we had street view coverage.  Our members have mapped around 1.7 million kilometers of roads, which is pretty awesome.  And that's actually just past 73 million images this morning.  And we have images from all around the world but 180 plus countries right now, and everything that uploads to Mapillary, we also have fully automated process.  We don't have to do anything on it.  It will actually be impossible for us to do anything on images.  On a good day, we'll be receiving up to 300,000 images on Mapillary.
      So this is just like a little zoom in to show you, like, how good of a coverage you have in Europe.  This is Germany, and you can see how good it is in Netherlands.  People have been crazy, and it's awesome to see how people are capturing street view.
      So here's an interesting picture.  To the left, you see the City of San Francisco.  And to the right, you see the City of Helsingborg in Sweden.  And the image to the left -- I mean they look the same.  And you see the green lines here.  This is where we have street view captured.  But in the image to the left, all of the imagery are captured by smartphone, by a free app that we provide.  While the images to the right, all of those images are professional 360 rigs that has captured all of these imagery.  So we do have full support for 360 too.  What's cool is that on the image to the left, the map to the left, that's all our members that have contributed.  They have been just driving around different roads.  But the map to the right, that's actually a city who has given us all their data.  Since we have this cool platform where people can just dump in imagery and most of those cities are open data initiative, and they want to give back their imagery that they have paid hundreds of thousands of dollars for to capture to the community, we have an initiative to upload images to Mapillary and host those images for free for the city.
      So what I want to show here is that it's the second thing; right?  We want everybody to be able to contribute to Mapillary.  It doesn't really matter what your intentions are.  Like, a city maybe has very different intentions from, like, what the private person would have.
      So now comes down to you.  How do you get your imagery up here?  Well, you can do it many different ways.  We have an app for iOS, we have an Android app, we have a Windows phone app, and what's pretty cool is that the iOS app, we have integrated action cameras.  Because all of these action cameras like a GoPro and the Garmin, we have APIs.  So you can just connect from your phone to these action cameras.  And what you see down here is actually a support of the Garmin camera has, you can connect up to four cameras into one iPhone.  So by doing this, you can actually create a really cool street view, and you can get, like, maybe full 360.  But if you want full 360, and this is what really really excites me.
      [Laughter]
      I mean after this presentation, I have to give out those at the door, so you can just pick up one of those.  This is actually our user from France.  He helped build this rig himself.  I think this is his Twitter picture, so this is really funny.  The cool thing about this, except that it's an awesome rig, is what he has on his head.  It's a 360 camera.  And I think that we are right now on the verge of getting really cheap 360 cameras out there in the world.  And that is prepared for that.  We have a super -- not super, but we -- yeah, we have a super platform to actually take care of all of these 360 imagery.  That's what I want to say about that.
      So talking about super platform, before I go into more about the platform, I want to talk a little bit about what's going on behind the scenes on Mapillary when you upload an image.  Because when we started out in Mapillary, we made a promise to ourself that we want every single image that they upload to count.  So if you upload an image now, and we do new technology in the future, we want to be able to make experience using that image even better.  So we've got to take you through a little what happens when you upload an image to Mapillary.
      The first thing that happens is that for privacy reasons, we blur all faces and license plates.  The image here to the investigate actually a license plate of a bus, I realize that's maybe not that obvious but a license plate to the bus and to the right, we have blurred faces.  So that's the first thing we do.  In the second step, we do a 3D reconstruction of the whole area where you have uploaded images.  So this is what you will see.  You will see a pleasant viewing experience.  But what we will see, what a computer will see is this 3D point cloud here.  So you're actually viewing a 3D reconstruction of everything that you upload.  And we do this adaptly.  So if someone uploads an image to a street and then somebody else comes back two months later, these two images would go into the same construction and the same point cloud.  What?
      [Laughter]
      You want to look good; right?
      So after this, we do semantic segmentation, basically what we do is we're trying to understand how -- what every object is in an image.  And here you can see.  We detected the road, we detected the pavement, we detected the cars, we detect traffic signs, and we segment everything out in the image.
      And then the third step we do is that we take this 3D reconstruction, and we combine it with a segmentation.  So what this basically means is that let's say that somebody takes 400 images down the street.  And in three of those images, there's a sign.  There's the same sign in these three images.  And many other platforms, they will show you three detections of this.  They will be like there is where the stood where I took the first image of the sign, and this is where I stood when I took the second and the third.  But what we do is that we combine these 3D reconstruction together with the segmentation and give you an exact location of where an actual object is.  So instead of getting three locations that is not true first time, you get an exact location for one sign.
      So how this ties into OpenStreetMap, I'm going to show you a little later.  First of all, I want to tell you that all of these features are available on Mapillary.com.  I don't think that the street sign feature is available yet because we rebuild our site, but it will be available from a few weeks from now.
      And I can talk for hours.  We just did a redesign of it, but you can go here, you can explore it, I think it's great, intuitive.  But what I want to talk more about is actually Mapillary as a platform; right?  We have all of this imagery and now you want to make use of this imagery.  And me as a lead architect, I always had this thought that, well, it's okay.  We can build your client for viewing images and doing this and that but if you're an OpenStreetMaper, we have to be able to let you build the tools that you want.
      So as a developer or OpenStreetMaper, we provide three different things information -- or three different segments or whatever you say in English to do this.  So the first one is our open source viewer, which is Mapillary JS.  I have to talk a little bit about that one in the next slide.  The second one is that we provide all of our spatial data, all of the extractions and everything we do as vector tiles.  And vector tiles is awesome because you can filter those, you can get everything out of them, out of there, and for the more advanced users, we also have an API where you can do stuff like blurring, you can do change sets, you can do stuff like that automated.
      So very important.  An open platform where you have a viewer and the viewer of course is open source.  We're trying to do everything we can to do open source.  The viewer -- I mean so if you go to GitHub Mapillary, GitHub.com/Mapillary, you will see that we have a lot of tools open source.  And just going to highlight two of those that we're particularly proud of.  And the first one is Mapillary JS.  And if you guys are familiar with leaflets or Mapbox JL, you can think about Mapillary JS a little bit like the Mapbox GL for street view.  It's an open source viewer that we built from scratch with no external dependencies, and it supports 360, you can make all of these different moments, and it's also component-based.  So you're able to add new components, do poll requests, you just play around with it.
      The second thing that we're super proud of is something called open SFM.  It's one of the most used structure for motion libraries right now.  And for those people who don't know what structure for motion is, I don't know that either.
      [Laughter]
      No, basically as far as I understand it it's basically tools that create three reconstructions in Mapillary.  And what it does is take all of these images -- oh, shit.  Five minutes.  And create a 3D reconstruction out of that.
      At least, but not important, Mapillary data is open.  It's shared by a CC by share like a license.  And -- this is important for you guys -- for OpenStreetMap, you guys have full data to derive from OpenStreetMap, OSFM contributor terms.  So every data we generate, everything from our tiles, everything from our Mapillary JS viewer you can see in there, you can take that, and you can put that back into OpenStreetMap.  Which brings me to the last part.  Photos and data for OpenStreetMap.
      So okay.  We have this big image repository, we do all of this cool processing on all of these images, and we have open platform that people can integrate.
      So let's look about what people has done in OpenStreetMap with that platform.  First of all, we have full access to all Mapillary images in JOSM.  So what you do is you go online, you install the Mapillary plugin, and then once the plugin is installed, you press imagery, you press Mapillary, and then here all of this green little dots here are Mapillary images.
      And what you can do after that is press one of those, and you get the instant preview of an that I come somebody has captured around the world.  It might be you, but you will see everybody's images in there.  And then you can use this image to update OpenStreetMap.
      We also integrated into the ID editor, the one that's on the Web.  So if you press this little button here, open up this little door, and you press this one here, you can actually see that you're getting the dots and the captures of all of the images from that street.  And what's cool here too is that you can actually also get all the traffic signs that we have extracted from the area.  And if you press one of those traffic signs -- this is a terrible image.  But you see down there where the hand is, there's actually a turning restriction there.  So then you can use that to update that.
      Well, there is no left turn here.  So a better tool for doing this is something that Mapbox has been working on lately.  What they do is that they take in all of our sign data from our vector tile, put it on the map, and then they use that to verify turn restrictions.
      So here you can see that I pressed Mapillary U.S. and a picture comes up, and you can see on the picture there's a turn restriction right here.  So boom you verify that and now we have better routing in OSM.  And that's it for me.
      [Applause]
      >> Any questions?  Any questions for Johan?
      >> Thank you.  Leonardo from Columbia, I'm a teacher in Columbia in school.  I'm teaching OpenStreetMap, and a few moments ago Mapillary platform we have captured a lot of places in Columbia.  And I want to know if this is possible to include the Mapillary app connected to improve the places, improve the -- for example, create the meta to improve the capture of the pictures.  Is this possible in the features?
      >> Oh, so you want the feature that tells you, like, basically --
      >> Help to map the road.
      >> Yeah.  You want to capture a city, and you want to, like, catch all of the nodes in that city.
      >> In that -- is it possible?
      >> We actually don't have that service you can do that routing.  But I think there are pretty good open source alternatives or services for doing that routing.
      >> Hi, quick question.  What dialing service do you use for the background --
      >> Sorry can you say that again?
      >> What kind of solution do you use for the server.
      >> For panorama?
      >> Yeah.
      >> So right now, we don't have a timing solution for panorama.  But that is something that we're working on to get in.  So you can get really super high definition panoramas.  Good question.
      >> Maybe similar to the previous question.  Are there standardization efforts underway to -- so from an API perspective, I wanted to get street level from data are there any standards underway that will help make this reusable?
      >> Not right now.  Not at the moment.  Right now we just have our API, and we have good documentation about how to use it.  So there's no, like, standardization process going on around that.  Not right now.
      >> Gotcha.  Thanks.
      >> Yeah.  I was just wondering if your JavaScript viewer uses HTML 5.
      >> Yes.  It does.  It actually uses WebGL 2 to do all of the 3D things.
      >> Any plans for node mapping?
      >> I mean the problem is always GPS.  If you have good GPS and let's say you have good 360 camera, it's easy to do.  One thing that's coming up pretty soon is a desktop uploader two where you'll be able to take, for example, panoramas in a house and then be able to place things on a map before you upload.  So it's possible, but right now I would say it's the GPS thing that makes it a little bit more tricky.  But we do have indoor -- maps indoor.
      >> More questions?
      >> The object recognition data that you guys are picking up, is that available in API as well?
      >> It's available as vector tiles right now, but there's also an API where you can get it from.  I would recommend to get it through vector tiles, though, because it's a little bit more cumbersome to get it through the API.  You have to send in the box and pan around to do that.  But, yes, it's available for our API too.
      >> Thank you.
      >> Any more?
      >> You said that Mapillary JS was open source.  I was wondering if that would work with, like, the lead JS plugin that blocks non-open source JS.
      >> That's a tough question.  I actually don't know.  I mean it's open source, you can, like, fork it and try it out yourself.  It's -- it's I really like that.  You should look into the code.  It's really cool code.
      [Laughter]
      >> We have time for one more question.
      >> Well, please thank Johan.
      [Applause]
      >> Thank you, guys.